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ABSTRACT ~ . . . 

ft Based on two, papers presented at the American 
Educational Research Association meeting 'in 1961, tnis 'report 
. provides an overview and major outcomes of a three-year study of the 
Research and development Utilization (BD0) .program. Ihe fiDU program 
tested and assessed a comprehensive model for assisting schools to 
use results 0 of educational research and development in local school 
improvement efforts. The first paper presents the outcomes of tjie EDO 
. progVaa at the school lev t el, while the second presents an, analysis of 
the vay in which product t characteristics, technical assistance, the . 
'internal problem-solving ^jcocess, and school" and poipil 
characteristics predict the level of success^ of the program. (This 
report is a summary of Volume 2 of the Final* Report.) ' (Author/MLF) 
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PREFACE 

■ 

This document presents an overview and major outcomes fr f om a* three 
year study of NIE's Research and 'Development Utilization (RDU) program. The 
intent of this^ summary is to erfable a wide audience in the, educational 
community .to srfare some of the important lessons, findings, and implications 
from that -program that may be useful in designing future dissemination and 
school improvement activities. The language of this report is somewhat 
technical and statistical , analyses are reported, A hon-technical suimnary 
is in preparation and will be available from NIE later in 1981, 

The RDU program demonstrated and assessed aj comprehensive model for 
assisting schools f tq use results of educational research and development in 
loQaT school improvement efforts. The RDU 'model is 'unique in combining 
several school improvement ^strategies into one program framework, permitting 
the combined. and complementary effects of the strategies to be seen. Seven 
RDU demonstration projects were in operation in various parts of the United 
States from 1976 to 1979* The assessment study, conducted by*Abt Associates 
Inc., endecj July ^, 1981. 1 

, This summary is based on two papers presented at the American 
Education Research Association Meetings in 1981* The "first gives an overview 
of the RDU program, the Abt Associates study, and i>rogram outcomes that 
occurred in participating schools. The second examines the contribution Jthat 
different elements of the RDU model , made to the program's overall success in 
fostering improvement in schools and in educational practice. 

This document provides a summary of one volume of. the Final .Reports 
written by Abt Associates .; t That volume focuses on outcomes of the RDU 
program at the school level. A companion volume examines the lnterorganiza- 
tional support structures that were established to help participating schools 
to (a) engage m^a systematic problem-solving proefess,* and (b) identify ^nd 
implement high quality products of educational Research and development that 
would be relevant ar>d usefcil in their local improve/rent effort* These and 
other ^reports from the „Abt Associates study, are described ir) an annotated 
bibliography at the end of this document. • * 
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. " * , .LINKING R&D WITH LOCAL SCHOOLS: 

A FEDERAL PROGRAM AND ITS OUTCOMES* 

The RDU Program 

In June oft 1976 f the National Institute of Education (NlE) estab- 
lished the Research and Development Utilization program as a new dissemin- 
ation effort to help schools .clarify and solve local problems. ' The RDU 
program emphasized a research-based, rational approach to local school 
improvement through the use of existing * validated products of federally 
funded research and development activities. This program was designed to 
achieve three major objectives: » t 
* • • * # 

• to Jielp schools alleviate specific, Ideally defined ' 
pr*oblems'<in the areas of basic? skills ancj career 
•education; \ ^ 

b « to help school and community personnel learn about 

the produpts t)f educational research and development; 
agd ■ 



• 



to increase understanding of how the local program 
improvement process can be better managed and become 

mrtrp f>Fff>nh 



mtire effective. 

* 

The RDU program was unusual * among federally funded dissemination 
strategies because it was equally concerned with the dissemination and use of 
t R4D prodocts and with the development of local organizational capabilities to 
Solve problems. Other federal programs have tended to concentration either 
product dissemination or local capacity building. 
• ■* # 

TheJtDU P rocess 
— z — "~~ ~~ * 

• The core of the RDU approach*was to provide each participating * 
site with assistance in. problem solving, broken into stages represented by 
the sequence oft activities listed below; 

• identification of a problem or set of problems:' 

• examination of alternative solutions to the problem, 

focusing particularly op the products of educational , 
RAD; k 

% 

^Election of a specific solution considered to be * 
appropriate to* alleviate the problem; 
• • • * * * i 

«■ m * 

V* • implementation of the solution; and 

, • evaluation anjj incorpo'ration of both the solution and 
1 the problem-rsolving process. " » 



■ "* T his Js'oction is. based on a paper presented by James A. Molitor, 
at the 1981 Actings of the American Educational Research Association. 




The service delivery system of the RDU program "operated through 
seven regionally* dispersed projects, each of which coordinated a network of 
organizations and individuals involved in the provision c?f services aqd 
information to local school districts. As a whole, the seven projects 
operated in 20 states and served over 300 schools or ^chool districts over a 
three-year period (1976-1979). 

The Northwest Reading Consortium (NRC) , under the overall direction 
of. the Washington State Education Agencj, operated as a consortium of four 
states in the Northwest: Washington, OAgon, Alaska and Idaho. The project 
built upon the existing Right to Read programs . in the four ^tates. (The 
JUght to Read program is a nationwide program sponsored by the U.S. Office of 
Education to eliminate functional illiteracy.) the Northwest Regional 
Educational Laboratory was also an affiliate, providing training to project 
staff* anJ support in the development of a" pool of R&D products. 

The Georgia State Department of Education operated "a project which 
provided funds and services to all participating school districts located 
in three .Cooperative Educational Service Areas^ The emphasis of .the RDU 
project in Georgia was on building local school district capacities in the 
early stages of planning and program selection. The. implementation phase of 
the problem~solving* model was subsequently carried out with federal funds 
available through the state department of education under ?itle "IV-C of 
the Elementary and Secondary Education Act and wath other state funds. 

The Pennsylvania Department of Education developed and cpordinated 
a school improvement process which involved the participation and resources 
of several organizations: Research for Better Schools (a regional education 
.lab) i Research and Information Services for Education (a statewide informa- 
tion and dissemination service); the Learning Research and Development 
Center at the University of Pittsburgh; and the State's Intermediate Unit6. 
The project's agencies were involved in helping sites with numerous defined 
steps, including a series of formal training sessions in problem solving at 
the school sites. m \ ' 

The National Education Association (NEA) operated its project in, 
collaboration with the state education agencies and corresponding state 
education associations in 12 states: Alabama, California, Iowfe, Massachu- 
setts, Michigan, Minnesota, Ohio, Pennsy lvama , Tennessee , Washington, 
Wisconsin, and Wyoming. "In contrast to ^the other RDU projects, tnis project 
Joe used exclusively on the improvement of teacher inservice education. 
Services were provided by two field agents in each- state whp trained l^ocal 
staff. 

Thp Florida Department of -Education served as, prime contractor in 
a linkage system which also involved the state universities (especially 
Florida State University and the University of Florida), and eight 'of the 
state's. Teafcher Education Centers (TECs). An important feature of this 
project was that training in group problem-solving techniques was provided 
not only to the*field agents (one of whom was located m.eachr. TEC) , but.'also 
to selected local school staff. \he school sit* facilitators, with the help 
of the field agents, were responsible for leading the staff^ at their sites 
through the entire problem-solving^process. > ■ . 



r \ 

The Career Education" Dissemination Project of the Michigan Depart- 
rent of Educati&n was designed to help -local sites meet the requirements of 
state career edteation legislation passed in 1974. One of the project's 
major objectives \w as *lo d6velop a- permanent ' dissemination and .diffusion 
system in career Vacation. Because of this emphasis ^on permanence, the 
project attempted Co work with existing structures and personnel m the 
state's intermediate sfchool districts rather than build new ones. The 
primary strategy was \o provide direct draining and prograrrtnatic funds to 
coordinators who were s\aff Members at local si tes . 

\ * * * 

The NETWORK IncOy a non-profit research and service organization 
1Q Andover, Massachusetts , coordinated a consortium of agencies in six 
states: m Minnesota, the\ agency involved was a teacher center associated 
with a university; in WashiiSgton, a local school district; m California, a 
regional education laboratory sponsored by NIE; in Kansas, an independent 
statewide education diffusion^ organization; in Connecticut, a cooperative 
service agency supported by loial schoM districts; and in Massachusetts, a 
division of the NETWORK itselfA This project was formed mainly to improve 
^the utilization of R&D products\in reading m selected local schools. The 
field agents provided assistanceX to the local sites, while a considerable 
^nount of technical assistance and support was provided to thd field agents 
by the project office in Andover 

y Some common features rag tlhoughout the seven projects and in thfe 
structure of support services provide^ to local schools: ^ 



\ 

\ 

i 



the operation of a project* headquarters to coordinate 
the services supplied to scnools; 

\ * 
the development and administration of a knowledge base 
composed of educatiopal reseaVch and development* 
products j 

the development of training 'and technical assistance 
components to serve the projectVs field agents and/or 
school staff; and " — * ~ 

the Uevelopment of project evaluation and re&arch 
activities to facilitate local se\lf-evaluation by the 
school' sites. 



RDU as a Dissemination Strategy 



To gam an understanding of how^ the ^ROU program operated, we first 
review 'the local school context within whicHf i£J>was working. In general, 
schools have been accustomed to one of two apprpaches to problem solving. 
One is a top-down approach in *which focal problems and solution strategies 
bSve been identified by building, and/or district-level administrators, 
sometimes with the aid <of consultants or other external resource persons 
called in from universities, labs and centers, etc. Although practitioner 
inputs may have been considered, decision making* was largely centraly^ir** 
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That is, whether at the local building/district level, or from some larger 
change program, administrators identified courses of action or programs 
to be implemented by local ,s<jhqol staff. Although >such decisions may have 
been informed by the latest valid research results, th^s research knowledge 
was not balanced ag^ainst--nor always cognizant of*--the wealth of craft 
knowledge resident in local teaching staffed Thus, this top-down decision 
making may not have been problem solving -in the sense of a rational, partici- 
patory set of activities as espoased^by the RDU program; rather it 4 was 
sirrply resource allocation by administrators faced with such* questions as 
wher<e to buy the needed new textbooks. 

• • *■ 

A second common .approach to problem solving may have been ev»en more 
prevalent. Tfus approach was characterized by the absence of .organized 
problem solving or change activities. Teachers were simply left on their 
own, "free" to identify and cope with problems as they saw them, perhaps 
negotiating with building or district administrators for needed resources on 
an individual basis. ^ 

These approaches to the dissemination and utilization of new educa- 
tional knowledge were significantly altered* under the RDU program. The new 
view of dissemination which RDU embodied incorporated a staff development 
process in which local personnel were to receive trairapg in problem-solving 
processes, enhancing their ability to play central roles in more broadly 
participatory rl bot tom-.up" decision makirfg.* ^ Working with one or more 
external resource persons who could link local staff to a large>*"l<nowlgdge 
base" (in the form of a compendium of educational products and materials), 
local staff would learn how to identify aqd prioritize then 7 problems and 
goals. They would then be assisted in a review and "screening" of potential 
solutions— the products and materials in the knowledge base which, when 
implemented, could alleti&te the problems they cho.se . to focus on. This 
assistance would inciuo&^rainincj in matching various characteristics of the 
products with the characteristics of the local problems and with available 
staff or other local resources. The goal of these activities was the 
careful selection of a product or program which "fit" local circumstances. 
Staff would then receive any necessary training (e.g., through lnservice) 
fot the implementation and\continued 'use of the adopted program. 

Implicit in this view of dissemination are longer range goals than 
simply 'identifying and coping with an immediate local problem. The training 
and involvement in participatory problem solving .and the increased awareness 
of prpducts were aimed at increasing individual teachers' inclinations .to 
implement and continue using adopted programs. But beyond this, the RDU 
program more generally was designed to enhance local capacity for identifying 
and solving future problems as veil. This is reflected in the fact that 
implementation and incorporation a product was only one RDU thrust; the 
other intent , was the incorpoi^tlmKoP the problem-solving process itself. 

Thus, as a dissemination and knowledge utilizatibn strategy, ; the 
RDU program involved mobilizing internal and external resources in enhancing 



*Note, however, that the "bottom-up" activities were* structured — 
sometimes very rigidly— by the project the school participated in, and H the 
problems ^ddrefesed in tHi? program werc-'restr icted to basic* skills and 
career education. 




school effectiveness. The "treatments" or "interventions" this program 
entailed included:- 

% H 

* — • " a large and accessible base, of educational products 
x which embody thfc knowledge to be utilized; < 

• a problem-solving process , including rational decision 
. making by a broadly representative- local action team 

(or LAT); 

• process assistancgfrom an array pf external human 
resources , lncludin^^ield agents , who connected 

* local action teams with* the R&D product basp and 
other resources ; and 

• limited financial support to some schools , usually 
in the form of small stipends to cover release time, 

, travel to demonstration or training sites, etc.^ 

The* Study of the RDU Program 

In .November 1977, Abt Associates Inc., a social science research 
firm based in Cambridge, Massachusetts, was contracted to conduct a study 
4 of the RDU program. The study addressed .six major issues: - 

• how relationships are managed J/etween various agencies 
which have the expertise and resources to help local 
schopls solve problems; 

% ■ . 

• to. what degree an intervention program sucrf as RDU can 
~ help schools overcome barriers to successful problem 

solving (such as limited access to information or lack 
of planning skills, etc.); 

* • to what degree the products of "educational R&D are 

relevant to the problems and contexts of local schools; 

t what the impact of the products of educational R&D is 
' once they have been adopted and implemented; 

\ * < 

• what factors contribute to the institutionalization of " » 
_ the RDU approach within a variety of organizations; * 

and ' 

• how field agents coordinate the flow oft external 
resources to schools, and whether this helps the 

— schools solve problems. / w 

- . ): % 

w As our earlier discussion of this # program suggested*, the R&D Utiliza- 

tion program was highly ambitious in its* aims. Not t)nly v/as it intended to 
increase teachers' awareness ^nd utilization of R&D products in local 
schools (product outcomes),, if was also intended to have a tnore global 
consequence; to improve the way schools identified and wfent about solving 
their problem^, both in terms of increasing the breadth of participation in 
the problem-solving process, and by making the problem-solving activities 
themselves more National (process outcomes). • I 

ERIC * \ * . 10 



Early in our study of the RQJJ program, we conducted a series of 
relatively brief familiarization visits to local schools participating m 
each of the seven operational projects. Our aim on these visits was to talk 
with staff of these schools so as to become more familiar with how the 
program operated at the local level. Through unstructured interviews we 
learned that, in general, awareness and utilization of new educational 
prodtfcts were, m fact, being increased through the use of improved problem- 
solving practices. However, it quickly became apparent that other things 
were happening at these schools as direct results of their participation in 
the program: the schools themselves were changing in a variety of ways 
(organizational outcomes), and* so were the school personnel (personal 
outcomes). This led us to expand the range of outcomes, to be studied under 
our research design so as to include an array of unintended organizational 
and personal impacts at least as important as the intended R&D product and 
problem-solving' process impacts. 

Data were collected in face-to-face focused but unstructured inter- 
views at 51 sites during 1978 to 1980. Case studies were 'written on 46 
sites, five of which also received site visits. We also conducted mailed 
-surveys of principals and a sample of teachers at participating schools 
during the fall of 1979. This paper presents an overview of the four types 
'of program outcomes we observed in our data. 

Outcomes for R&D products . A major objective of the R&D Utilization 
program was to install an appropriate R&D product in schools participating 
in the program. Thus, the degree to which schools identified, adopted, and 
implemented a v product relevant-to the problem they sfaught to "alleviate is a 
critical measure of the intermediate &v proximal Success of the program., 
Qttfer intermediate product outcomes include various aspects of teacher 
satisfaction with the products, the numbers of_ pupils and the percentage of 
their school days affected by implementation, an"d~h"ow difficult the product 
was to implement, including the neetd for adaptation. 

Approximately 100 , different products or sets of curricular materials 
were adopted by the participating schools. The most popular products, in 
terms of frequency of adoption, were such reading packages as the Wisconsin 
Design for Reading, Exemplary Center for Reading Instruct ion (ECRI), 
Houghton-Mif f lin Basal Management System, and San Diego Right-to-Read . 
Career education packages which were most frequently adopted included Career 
'Development Centered Curriculum, It Works, and AEL^Career Decision Making 
Program. Popular mathematics programs includecU^Brevard County LAMP and 
STAMM. In general, products ranged from lists of objectives for teachers to 
detailed Management programs; some included variety of materials for 
classroom use, such as* slides or filmstnps and tape cassettes, student work 
and record-keeping sheets, and associated texts. ~ 

The characteristics of the products themselves *varie # cj along a 
number of dimensions in addition to whether they were R&D- or practitioner- 
developed* For example, some were intended for use in only one classroom, 
whereas others were implemented throughout the schools. Some, such as San 
Diego Right to Read, consist of sets of ideas from which adopting teachers 
may pick and choose, whi le .others, such as ECRI, require significant, 
highly structured changes of all tfeachers and are therefore more difficult 
to implement. 



1 ■ 

< Of particular interest her6 is the fact that the products ancj* 

materials adopted were mop* frequently practitioner developed — i.e., NDN 
product s~tfcian R&D-based materials such as those in the NIC catalog. This may * 
be due to a conspicuoqs dearth of R&D products in some areas, such as career 
education. In this case, interest only .burgeoned in the e^rly i97Qs, so the 
time available for producing a variety of relevant, validated products in 
time for the RDU program (which began in 1976) was too short. 

|V Other areas in which validated R&D products were scapte inclJde 

school- or distnct-wide planning, inservice training, -and basic skills at . 
the secondary level. In this latter case, the need for products—especaally 
reading at the secondary level—was not recognized until after the RDU 
program was underway. 

• • *" 
Finally, some schools 1 needed assistance with topics stemming from 
racial and ethnic integration and the special needs of minority groups. The 
available pool of products for bilingual students was relatively sparse. 

Practitioner-devel&ped products also had a logistical advantage 
m that frequently there were experienced trainers, funded through such 
other federal programs as NDN', who could provide pre-implementation assis- 
tance and follow-up; service to adopting schools. As we will see in later 
analyses, availability of training in product use was strongly related to. 
several measures of program success. 

* * 
Within the schools that ftad reached the "product selection" stage by 
the time of our final data collection, over 80% Of the teachers responding 

' to our survey indicated they were using the product or had used it iq the 
past. Another }% had 'definite plans to use it in the future. Fewer than 

j 202 of the users reported the products needed adaptation to a great or very 
great extent, and their use was at* a high level: over 65% of those using 
the products reported they used them with all of their students, and 85S of 
the users stated the product was regularly used * at least once per week. 
Satisfaction with tfie adopted products w&s' generally high,, with over half bf* 
the users reporting that to a great or very ,gr6at extent', the products were 
directly relevant to the most pressing pr6blem in their school, met a need 
in the classroom, and provided new idea£ and not just ideas they were 
already using. Another 25%-30£ of the users reported these statements were 
at least true "to some extent/ 1 

The users did nob encounter serious problems with implementing 
the products they adopted. About 20S reported the products required major 
, changes from their previous teaching style, changes in classroom organization 
or management, or substantial additional record keeping. Qnly about 9% 
reported difficulties in implementing the program pr materials ,to a great or 
very great extent. However, we will see in later analyses that perceived 
difficulty in implementing adopted materials was positively related to 
school-level outcomes, .possibly because the greater investment of effort 
enhanced feelings of program ownership: 

• * i * 
<f more long-te#m, or distal, product outcome is the extent to which 
it is^ incorporated into, the everyday functioning of the classroom— i.e. , 
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the extent to which product otilizatiop is "routinized ." At thisr point 
dat^i from two sources 'become relevant: in ordqr for the product to be 
incorporated or r&iinized, npt only roust the teacher^ indicate they plan to 
'continue using the program or materials in the future— albeit with modifica- 
tions—tut .building administrators must indicate that certain steps necessary 
td^Qftsure the ^continued possibility of use have been taken. Thus, although 
. 83%. of- the users reported they would .Continue tp use the products, we must 
* still consider such long-term questions as whether, the product had been 
"incorporated into curjuculurn plans, measures taken # to ensure that new- staff 
wotfld use the product, etc. 
■ " ■ 

' Building principals of, $choQfe which had adopted products were 
asked whether a va&iety of such eyerrtar hati already occurred or would xJe fi- 
nitely occur in the future/ In 6ver 70S df. the schools, the products had 1 
been or* would be incorporated into purnculym plans. About # 50SS reported 
that written guidelines, for product use had already been developed, and 
another 115*. reported tftis would definitely occur* Almost £0% reported that 
. r*ew staff wouid^ receive training or^onentation. in *the upe of the. products, 
and that training *or inservice .for current t staff would t be used to ensure 
.continued product utilization* Qvei? 1 90*Kreported that.som.e of all of their 
teachers would use the products t(\ some extent , 62% indicating the products 
would b§ used quite ^extensively. * * . r 

Outcomes for the problem solving process* In addition to the 
emphasis on getting an R&D product installed at participating sites, a major 
focus of the R&D Utilization program was tb'^ncrealse, a *scHooVs capacity to 
deal with its problems*, by providing staff,, with^ trai^rig and pTacticQ in 
group problem-solving processes. Though not explicit in any RDU project, it 
+ is implicit that there are two critical a$pet?ts to this goal of improving 
problem solving at the site level; one' involves the us** of a rational pro- 
blem-Solving model , while the other stresses the need for relatively broad- 
based participation ,in problem«Tsolving activities. That is, any and all 
groups which will be Effected by the decisions, reached should .be represented 
o*n-the problenwsoLving teajn. Thus, the exfSnt to which the sites actually 
used a rational pjjfblem-solving model, and . the* extent to which there was 
.broad partiqipat ion in problem-solving activities become two important* 
intermediate outcomes s)f participation in the RDU program, *Note that both 
could vary Ifoc each site across stages 6f the local process. ^ 

• As the f>ra0ram operated at the sit£ level, these two goals were 
generally met as j/e site went through the problem-solving^ process: in most 
f ^ases, a field agent was available to guide the site's activities, and in 
some*cases economic sanctions could be applied should the frite not ""toe the 
marie. 11 But. factors inherent in the* prop ess* militate against its later 
replication (such as the fact that the process is complex and time consuming, 
which some staff, resfented very/nuch). Our conversations with site staff also 
revealed t that even where they felt tfiey could go through the process again 
without the aid of the field agent, t£p .release time provided by the RDU 
program was often a sine qua non of its success: otherwise teachers could 
not spend the (often substantial) amounts of time the j>roblem-solvihg model 
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required,* Vte must also t remember that in general , %iproving their problem- 
solving practices was not the reason sjutes got involyed. in this program. 

This implies that £*more distal process outcome, was the extfent to 
wbich the improved problem-solving practices— or at least some of them— were 
likely to be used again in deal Jng. -with other problems. ' * 

r » * 

In terms of' breadth of participation ^ the problem-solving process, 
our data suggest that there was generally good > representation of groups 

'whi&h wouttt ultimately be affected' by the decisions made. This was true f 
across all stages of the process, although # shortcomings other than broad 

, Representation were evident. In the ^pase of*90 sites on which we had highly 
detailed data, we rated them" on Jtheir problem-solving ancf grpup decision- 
making activities in terms of a lifting we developed oT^desired traits 
specific to each phase of the process. Where a site's rating was reduced, ' 
we indicated specifit types of deviations from the£e traits. These fell 
into interesting patterns across the various stages. 

t i 92% of the sites, problem-solving teams were established, and 

there was generally good representation on these- teams of thelgroups 'which ♦ 
would be affected by the teams' ijgcisions. However, during the early stages 
of problem .solving (problem Identification and solution selection), we 
found that decisions were often made or heavily influenced by administrators 
or' other external parties* This ^as true during problem identification at 
}6% of the sites, and during sblution selection at 2k% of the sates, lri 
the later ^tages of 'problem solving (planning for implementation and imple- 
mentation), the continuity of formal decision-making groups was not always 
upheld; this *was*rtrue "at almost - 20% of- the sites* teetings became less 
regular, at 26% of the sites, and during planning for implementation, deci- 
sion-making at 21% of the sites/Jid not involve all affected groups. 

In terms of rationality * of th« process, we found that although 
many sites appear to have adhered closely to the principles of sound problem v 
solving, well over 40S of the sites showed at least one—sometimes severaK- 
departures . from t our ideal criteria. During problem identification acti- 
vities, the most frequent* variant was that the problem definition was merely 
*a restatement of someone's a pnflfri assumptions' or pet tfaeory (46$ of the 
- cases). A AUertiafaive, definitions were not posed^ and considered (43%J > and 
the problem w ^aH»j^ adequately specified prior to heginning the searoh for 
solutions, ( JA?™^^*™ 





*0uring solution selection, the most common deviations were that 
alternative solutions were not carefully examined according to a . set of 
explicit* criteria (44S»), and qyidence of solutions' effectiveness or •suit- 
ability was not obtained. (32S)\ .During planning for Implementation, kk% 
of the sites did not make formal plans for some or most aspects of imple- 
mentation, v t ^ 

*Key characteristics of this model are (10 thorough analysis and - 
prioritization of school needs or problems before searching for school 
Improvement strategies* (2) a search outside the local school system for 
assistance and information, particularly in the search for a6lutions to 
problems;, fj) systematic examination of alternative solutions according to 
explicit criteria; and (4) a focus on solutions whixsh havSSbeen field tested 
and empirically validated. . 

14 • ' 



During the implementation stage, adherence to sound Jprattice' was 
generally much closer, with only about a Quarter of the sites showing any 
deviations. Most common among these deviations were*' not taking adequate 
measures to ensure implementation 0/ essential features and goals of the 
prodiJbts (23*), and adaptations 0 f the products implemented (235S)* which 
might not**have beer? necessary. * ■ 

The most distal process outcdme measure is the extent to which 
the sites repeated *or intend to Repeat some or all of the problem-solving 
process to solve other school problems. Our 'data showed that 41% of the 
principals and about 34* of the teachers at participating schools said they 
had repeated (or wej*e repeating) all or part of the ROD approach to address 
another issue in their school. The most often repeated part of the approach, 
included use 6f teams of teachers and administrators to make decisions. 
Schools we^e less likely to report that they would use the services of field 
agents or other external consultants. 

Outcomes for participating schools as^rganizations . We have seen 
that* the R&D Utilization program's objective of getting products installed 
at participating sites was, in large measure, achieved, although many were 
JJfltl R&D products. To a Somewhat lesser extent, the program* s *goal o*f 
lrapFovwig local problem solving was also achieved, at least fbr this' one 
-£ime. TOwevec, neither of these categories of outcomes necessarily implies 



that there will be any enduring changes in the schools as organizations. 
y That is, the simple fact that a certain set of activities vjas accomplished, 
culminating an the adoption and implementation of, for exemplar a ne * 
reading program, need not mean that the school's curriculum was improved, *or 
that the new materials were in any way better than those used previously. 1 
Similarly, the organizational structure of the school, which is difficult to 
change under any circumstances, can ^survive other changes without alteration*. 
After allS organizational change^ were Tiot the intended outcomes of the RDU 
program.. * * - "•-*,- • 

However, as we quickly learned during our preliminary site visits, 
a number of unanticipated effects were occurring on the school's themselves 
and on the'ir staff members. The spontaneous reports of such effects by 
teachers ancj principals in unstructured interviews led us to ^develop speci- 
fic line9, of. inquiry into these organizational and personal effects. Askiftg 
teachers to serve as internal observers £f what wa£ taking place in their 
schools,' we found that of the teachers .said that^ the following 

factors were somewhat or much better at theif schools; improved "curriculum, 
better materials available, greater collegialiiy .among staff, and generally 
better teaching. About 40* of the teachers reported school organization ar 
'management , decision-making and problem-cpl-ving procedures, and morale Here 
somewhat or much better. >About 45% of the teachers said Jthe image of ^Jtneir 
scfhool in the -community hadlbeein somewhat or much improved. 

To be sure, 30S-50S! (\ the teachers Reported "no change" on any one 
of these dimensions, but Qnly^a tiny minority (generally fewer than 2% pf 
the respondents, > said these dimensions had been % affected adversely. Com- 
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*Later analyses showed t^afc^TTocal adaptation of .the products was 
negatively related tp program outcome* measures. ■ , 



parable data from principals of participating schools and from our research 
teams t ' visits to J$he schools confirmed these reports** , 

Outcomes for Participating Staff s % $s a result of their participa- 
tion^ the^ RDU program, the staff of the schools involved had a variety -of 
experience^: some received t paining in group problem-solving techniques; 
some had the' opportunity to visit other schools or educational product 
developers to oDserveflttD products in usl. Others received training in jbhe 
use of aqr-adopted product and returned to their schobls.to train their 
colleagues^ still, others became spokespersons who visited other schools to 
tell of then; oytn* experience with using a new* educational product. 

An anonymous questionnaire was *(jsed to ask participating teachers 
about the extent whiCh they personally benefitted from involvement in 
I the RDU program n\ a variety of ways. In general , ' 155S-3055 of tKe teachers 
-reported they had benefitted in the following ways to a great or very 
great extent: their teaching skills had improved; leadership skills Had* 
improved; theV had learne.d about curriculum development \ had more selV- 
confidence and new resources ^fer^ffelping their colleagues. Another 30S-4ftS 
reported these, benagts "to'some extents Increased self-confidence and Jc 
satisfaction *eM also Reported* by 45S-505S^pf the teachers, and nearly X 
reported they bad been *giv$n increased* responsibility or been promoted do 
some extent or to a. gfeat or very great extent. % 
« * 

A Model For Examining Impacts of the R&D Utilization Program 
. \- " • . ' ■ 

As vie saw in the preceding bver^iew of selected program impacts, 

She available outcome data 'are ex tensive-- too extensive, in fact, to .permit 
s to analyze all 'of 4 our variables. To reduce the number -of outcomes to a 
more manageable 'set for analyses, we developed a number of summary additive 
scales. In this section of the report, we identify the qatcome measures and 
present data snowing hew^fcJheSe scales are related to each pther. 

" * ^ • 

The outcome measurelk.developed include the following! 
» * " 

Process* out comes 

« • Site -satisfaction with the problem-solving process , 

baped on reported- satisfaction with the services or . - 
aqtiv-tties of the local action team, the field agentf, 
developers of adopted materials', and the an>ount of time 
require^' t*o complete the process; ' 

- * • Sitevsatfsfaction >with the activities of the field 
agent, including the field agent's assistance «with 
various ♦aspects of the problem solving process such** 
as 'diagnosing the problem, developing criterfa for 
% selecting a solution, screening potential solutions, 
locfetiqg additional technical resources, etc.; 

: , - 

*Tha Pearson correlation between principal reports and our field 
teams' report of 'organizatdohal impacts was .44 (p<.01), and between our 
field team artd teacher , reports was .*55 (p<.01). 



• Irtcorporation of .the problem solving process , 'sucft as 
, reuse pf all or part^qrf the activities and procedures 
which the process Jrivolvedj , 



^ r 

Product Outcomes 



■ Extent to wtuch principal and teachersfreport the 
problem has been solved*throuqh use of jfche adopted 
materials , including lipprovemepts^ih ^pupils performance , 

, , # * attitudes, find behavior; and 

■ ^Incorporation of the adopted product and/or materials , 
, - a measure of the extent to which use continues after' 

implementation. 

v • ^ 

Organizational flu t comes. ♦ f 

Impacts on the achool as an organization , a global 
>' measure of lippact op the school including improvements t - 

(as a result of participation in Jthe RDU program) in . 4 

T curriculum, materials, school organization, staff 
morale,* etc. , 

* Outcomes for Participating' Staff <+ * 

t ^ Personal impacts on participating staff , including 

improved teaching, enhanced leadership skills, promo- s~ 
* ^ tion? or ^increased responsibility, etc. 

We expected that these measures would be interrelated in ways which 
wouid suggest a model* for examining program impacts at the siae level* 
For example we predicted that more distal outcomes such as incorporation of 
the adopted materials and the proceas* would be a result of more proximal 
or' intermediate outcomes such as satisfaction with the process, satisfaction 
with the field 8gent,*ahd so forth. ; * 

To investigate this matter, we performed a series of stepwise regres- 
sions, using; each distal outcbme fls a dependent measure* with the others as 
predictors. These regressions are summarized 4n Table l r which presents 
standardized regression coefficients for those variablea entering ss pre- 
dictor^ and\ncreas ing the R (proportion of explained variance) by at 
least 1%, along with an indication of their order of entry*. The selection 
process^ fas stopped when no further variables met this criterion.. raw 
correlation matrix is presented in Table 2. 

To graphical 1 y summarize bow thege outcome measures seem to he 
tied 4 together, we -p resent Figure 1, which j& a schema of their interrela- 
tionships suggeste<ri)y the regression results* In this figure, note That the 
outcome measures ^° t^e * e ft of the diagram are those assumed to be more 
immediate or proximal, while those to the rjtjht are as'sumed to pe more distal 
outcomes. He will first discuss the model in brief ,/ then return to examine 
'the implications for the distal outcomes.' f f 



# Other distal OutQAmes include a M Oxtent to*wh*ch the problem is/ 
solved," personal impacts, and organizational' impacts. 
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Table. 1 '■*'**, , 

STANDARDIZED STEPWISE REGRESSION COEFFICIENTS 

FOR DISTAL OUTCOMES ON .OTHER* OUTCOMES • , ;. 
(N ? 179 schools) 


* 

• 

V A. 


Other ' ^ 

Outcomes 

*» * 

• 


♦ 




Distal'Outcomess ' 




* u Problem 
Solved 


Incorporation 
of R&D Product 


Personal • 
Impacts 


Organization 
Impacts 


Process t 

, Incorporation 


Satisfaction with * 
Problem- Solving Process 




.14* 

(3)a " ■ 


m 


.16V 
.(4} . 

4 


-.01 
(3) . 


Satisfaction with 
Field Agent 


* * 


* 


(3) \ 




i 

» 


Scope of Implementation 


.19* 

(4) 

« 


.32** . - 
(2) 




■r22»* 

.(2) ' . 




Problem SolvetJ m 
§ 


* 


.26** 
(t) ' 


.29** 

'(2) 


.27** 

(f) 




Incorporation of 
Products 


/ 

.25** . 

(2) 


/ 


* * 




.12 

(3> 


• 

Personal Impacts 


-.22** 
(3) 






J9** 




* 

Organisation Impacts 
4 


- .32** 

(1) 


4 


.33** % 

(1) 




.25** 
(1) 


Process Incorporation^ 


4 




i 


.16* 

(5) 




. Multiple R?: * ♦ 
. ■ 


■ ' .48 


• J5 


.32 


.51 


• .16 


Adjusted R 2 ' / . 
* 


x .47 


.33. ■ 


.30 


.50'* ' 


.13 


Number in parentheses indicate's order of entry in stepwise regressions. 
* P i .05 
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Table 2 

PCAKSOHIAN OORftCLATlONS AMONG OUTCOC HOJSURES 
(H z ICLO^cSools) 



u 

•. . J. 



Outcome He 33 urea 



Satisfaction with 
Problem-Solving Process 

*' * 
Satisfaction with 

field Agent 

Scope of Implementjatifjn" 

* • f 
Problem Solved 



Incorporation or 
ft&D Products * 

Personal Impacts 
■ 

'Organization Impacts 



Agent 



r Product 



y 

Sat 15 feet loo with Set is feet ion Scope of Problem .Incorporation Personal Organization • Process 

Problem Solving with Field Impl erne ntit ion Solved of RAO k Impacts Impacts* incorporation' 



.30" 



,35" 
,49" 



,J4» 
.12 



^50" 



.30" 
,27" 
,29" 
,48" 
,24" 



,43" 
,26" 

.60" 
,jf6" 
.49"* 



♦27" 

4 

* 9 • K 
Ml* > 

-27" » 

* 

.28" 
.21" 
.39" 



•• p < .Ql 
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* * FIGURE 4-1 

Schema of Outcome Measure Interrelationships 



Satisfaction with 
Field Agent 



Scope of 
Implementation 



^ Personal Impact 



Satisfaction with . 
the Problem-Solving 
• Process 



Extent to wh 
Problem Solved 




"Impacts on 
School as 
Organization 




The most uimediate outcomes ift this model are those which are assumed 1 ' 
to occur closely on the heels of selection and implementation of the adopted 
R4D .product. These include two* process outcomes: satisfaction with the 
activities of the field «§gent, and satisfaction with the problem-solving 
process, along with one product outcome: scope of implementation of the 
R&D product (which refers to the proportion of pupils in .the school who 
are exposed to the adopted product and the proportion of their school day 
affected by its use). 

An intermediate outcome, H and one which our analyses suggest is 
strongly .related to the -distal results, is ^mother product outcorfie, the 
extent to which site staff report th& product has solved the problem, Wot 
surprisingly, this outcome is strongly related to the scope of product 
implementation, and is a strong predictor of a third prctfyct outcome, the 
extent to which the adopted product is incorporated. Product incorporation, 
a primary aim of the RDU program, .is also related to the scope of product 
implementation, and also to reported satisfaction with the problem-solving 
• process. 

Problem solution was also strongly related tp two outcomes we have 
identified as "spinoffs, 11 since they were not really the intended cQn- 
sequences of the RDU program. The first of ttese is a staff outcome measure, 
reported impacts on participants in the problem-solving process. This global 
measure' includes* reported improvements in dreas such as teaching \skills, 
leadership skills, or morale, resulting frojm having gone through the RDU 
-process. The extent of personal impact^ was also strongly related to reported 
„ Satisfaction with the activities of the field agent, with whom the staff 
worked during the program. I 

The second spinoff effect of thelRDU program was also strongly 
related to the extent to which the problem Iwas solved. This organizational 
outcome was the global measure of impact pn ttee participating school, and 
includes measures of improved curriculum and materials, decision-making 
structure, staff morale, and the school's iftage in the community. Since thg 
Organizational impact measure includes staff morale, it is not surprising 
that it is qlso related to, personal impacts' on participating staff.* 

A second primary aim pf the RDU /program, along with incorporation 
of the adopted R4D product, isy incorporation of the improved problem solving 
protest into school and diStmct decisionmaking activities. That is, the 
RDU program intended that thi rational, participatory decision-making model 
it aspoused would be utilized! again by the sites to address ot^er problems in 
the future. Based on our visits to over 50 participant sites, we consider 
this we most distal prograrrk Anpact. It is most strongly related to the 
global measure of organizational impact, end to incorporation of the adored 
R&D<proVuct (the other pramaiy aim of the program). Predictably, incor- 
poration\of the process is also related to satisfaction with the process. 

. But incorporation of nhe process proved to be difficult at the 
site level, and for this reason we Suggest it is the last outcome .of the 
RDU program to be achieved. Our site visits strongly indicated that th$ 
* problem, with process mcorpgration lifls in the nature of the process itsfilf: 
it was complicated, time consuming fto the point of frustration for "many 
sites, only poorly unc|£r stood even by many of it& participants, and not a 
major site goal. Even at siteV where there were clear indications the 
process (or part of it, or something .like it) was being used again, members 
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*kf the local decision-making team expressed confusion over^ what they were 
recall/ doing: "Why are we repeatedly p rirt|t>h ,^hq " Wh y "™ um — 

spending so much time on this survey of the community 7 " * "Why can r t you just 
show us sojne products and we'll pick the one lifc&best 7 " "1 just can't 
look at another reading progrd>H— they 're all alike, anyway 1 " Finally, even 
in many sites where staff reported they understood what they were doing they 
were candid in admitting they could not do it again without the help of the 
field agent (or some other external human resource). Since incorporation of 
the process was a pntical thrust of the ROU program, its elusiveness is a 
major" probleft to which we shall return in a later section when we investigate 
the efficacy of various aspects of the RDU "treatment" in producing impacts 
qn sites. 

• ■ 

The implications of our model for incorporation of the adopted 
progrqp, on the other hand, are more clear. Incorporation is more likely if 
the product solves* the problem,* is widely implemented, and selected via a 
process wfuch does not alienate participating staff". a More specifically, 
a decisionmaking process which ensures a Close match _betpeen the character- 
istics of the product and the problem i± is to address, followed by wide- 
spread implementation of that product increases the likelihood of later 
product incorporation. 

In the case of personal impacts on participating staff, it is not 
surprising that implementation of a product which seems , to alleviate the 
problem would enhance teachers' feelings of. classroom efficacy. Similarly, 
interaction with a competent field agent during the complex problem-solving 
process is likely to increase awareness of R&D resources, bring out leader- 
ship skills* increase interaction with colleagues, and enhance morale as a 
** result. In Louis, et al., (1981) we examine factors related to satisfaction 
with the field agent, and we will see the importance of product character- 
istics— R&D products being a critical part of the* ROU "intervention" — for,, 
Site outcomes. * * 

, Finally, we /nay briefly consider our model 's impficatioos foj organi- 

zational impacts. These appear to be enhanced by the effectiveness of 
the product in alleviating the problem, the scope of the* product's implemen- 
tation, dnd the magnitude of the personal impacts on participating staff just 
discussed. Our analyses suggest that .organizational impacts are also strong- 
. ly affected by the characteristics of the adopted product and by other 
* aspects of the ROU intervention! 

Ihe Range of Site-level Impacts of the RDU Program 

In conceptualizing how best tp measure program impacts at the site 
level, two basic options were identified by our project staff. One is a 
straightforward empirical approach which involves ttte development of a 
variety of scales from batteries of items in, the surveys of principals and /• 
% teachers 'and from the coding of case Study and site visit data. A second, 
more typological approach was suggested by^ our increasing familiarity with 
tbe sites' experience, gained through site visits and through the coding of 
site, visit and case study data. This second approach mdkes'useof more 
global assessments of the kinds of outcomes we saw, and is appealing because, 
it reduces the number of different dimensions of program 
> „ in some of our discussions. « 
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To develQp a typological outcome measure, we focused on four measures 
\)f program impact: incorporation of R&D products; incorporation of the 

* -problem-solving process; impacts on the schooT as an organization; and 

personal impacts on the staff at participating-sfchools. CThe computation of 
these and qther measures of the RDU program's effects is discussed'' in the 
v technical appendix to the final report.) Note that these include measures of 

• . the two primary intended;, impacts of the RDU program—incorporation of R&D 

products and the pr obJem-s6Tv J mg process--and the two areas of spinoff 
^effects we observed — organizational and personal impacts. We 9 feel that the 
following typology captures the range of global oOtcoaes found at the sites 
included in our analyses, and provides a concise summary of the RDO program's 
success. Sites were assigned to categories, on the basis of whether they were 
"high," "moderate-U-low," or "low" on the, four impact "measures cited 'above. 
'* The resulting "categories are: 

• LarQe-scale RDU success characterizes sites whnch 

\ generally followed the RDU model for problem solving 

with a great deal of fidelity, implemented en R&D ,. 
product from their projects knowledge base, and showed 
unmistakable signs of incorporation of both the product, 
and the problem-solving process, along with\such spinoff 
effects as personal impacts on'participatmg. staff and 
impacts on the, school as an organization; (?34% of the 
1 sample) 

• Mixed success sites were those which had two high 
ratings, one being a program goal (either product or 

- process incorporation) and the other a spinqffyoutcome; 

(17K of th£ sample) " 

«, * 

• RDU success characterized those schools which had one or, 
) two high scores on program goals; (16% of the sample) 

• Spinoffs were those sites which had some^positive • 
♦effects qn the school as an organization and/or personal 

* * impacts on participating staff, but whi'ch*did not adhere 

' closely to the problem-solving or product adoption goals 
of the program to aTiy great extent. Note that in many 
cases, schools in this category had their own agendas to 

» begin with--e.g.\ developing curriculum guidelines — and* 

used the resources of the RDU program tc^achieve them; 
*(1(& of the sample) 

• Moderate to low Success characterized -those schools 
which had moderate to low ratings on 3 or A outcome 
area?, anjj no high ratings at all; (10% of the sample) 

• .* 

— • failure .c haracterized those schodls which were very 
poor achievers on two- or more outcQme dimensions, and 
— — - which had no high rating^; (13% of the sample). 

The val'idity of this categorization is supported 6y consistency 
with^t^her findings, both related to other outcome measures and to what was 
known from the study of the seven operational contractor*. For example, 
examining how other qutcome measures were distributed amgng/these categories, 
we .found that sites classified as large-scale RDU successes also showed the 

> ■ \ 
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highest averages on measures of the scope of R&D product implementation , 
reported that thp problem they were addressing through their RDU participa* 
tion was solved to the greatest exb^nt, and reported the highest levels of 
. impacts on pupils. These sitnn aLro showed the highest mean level of satis- 
faction w^th the problem-solving process , and were the most satisfies with 
the activities of the field agent* In addition** they had the highest per-' 
ceqtages of staff, reporting that RDU was quite different from previous 
problem-solving practices. - " ~* 

* *■ ■ 

y Sites classed as very low successes^ as discussed above, show very 

Jrcw problem solved scores and indicate the lowest levels t of satisfaction with 
the problem-solving process and the field agent. Staff at these schools 
also indicated that the *RDU program was not vfery^ different from previous 
problem-solving activities, . ' 

Summpry and Conclusions . v 

We have seen in ^this chapter that the RQU program appears ,to have 
had a variety, of positive impacts, not only iq itSi. intended area^ related to 
the use and incorporation of new curncular products and materials and to the 
use of an improved problem-solving process , but also in two areas we identi- 
fied as spinoffs* These later impacts included .positive effects Qn partici- 
pating staff and on their schools as well. We have also seen that the 
various outcomes we identified may be interrelated in ways which provide a 
model for examining program impacts* 

What we have not yet seen is evidence that the various elements of 
the RDU treatment" — the products, the problem-^qJ ving process, and the use 
of external human resources — are directly related to the magnitude of these 
.effects* This identifies the next phase of our analyses, presented in'tfie 
follpfting section* 

y - . 
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PRODUCT, PROCESS AND PEOPLE IN THE R&D UTJLIZATION PROGRAM 
THE POWER OF THE INTERVENTIONS* * J* 



Introduction 



i 



The objectives of^fc*ris paper are to examine the effect of'tffe 
strategies utilized in the R&D Utilization (RDU) program on school improve- 
ment outcomes, and to further examine the relative . power of the RDU "inter- 
vention" aa compared to the 4< hon-manipulable characteristics" of the schoola 
involved. 

} The RDU Strategy: Where It Fits' Into a Larger Polic y Picture 

: : - : ■ — ; : . 

There are severs! basic. federal/state roles that support local school 
improvement efforts: - . * , 

• Legislative/aidministrative mandate ; This strategy 
/ involves developing laws and regulations governing 

minimum standards for staffing , programs, or even 
student achievement. When accompanied by effective 
'sanctions, it, has been viewed by sOme as the most 
efficient-tplthough not necessarily most effective — 
means for producing massive local chartge. 

fesource support : The resource strategy provides 
fcpsit^v^ incentives or assistance to districts that 
wish to engage in school improvement activitiea. * 
Within the resource strategy there are three distinct* 
types af support: . % 



-fiscal strategies , whidh may take the # fprm of "seed 
money" (temporary funding fpr improvement activities) 
or mote permanent fprmul* funding such as Tjttla^; 

— technological strategies , which support ftateriala • - 
- and program development, and make information about 
new practices available; and * . - 

— process/people strategieB f which .support free or 
very lftexpensitfe technical assistance, training, con- 
, sultatioti or other human resources. 

The major federal strategy in supporting school^ improvement has been 
a combination of direct fiscal support through formula fuOding *f various \ 
types, combined with legislation and regulations which require many, if not 
„ Aast, districts to make changes in their curriculum, staffing, uffe of time, 
apape and facilities, and other areas^of $chodl functioning if they are to 
receive federal funds. The RDU strategy looked quite different from this: 
it emphasized voluntary involvement, offered small amounts of seed mooey 



*This Section is, based on a paper presented hy Karen Seashore Louis 
and Sheila Rgaenblum at the 1981 meetings of the American Educational 
Research Association. \ - A 
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fundihg, and put ^ major emphasis on providing both technological and 
process/human support that would be responsive to locally defined vteeds. 

Stimulating Voluntary Change* in Schools: Apgutpents Against the Effect- 
iveness of Small Sca^e Externa]»Intervention * 



Although »the ROU Program involved a rather heavy level^of effort on 
the part of loeal/ school personnel,, it was in large measure an external 
intervention. There is an accumulating literature, however, that suggests 

that local school .llttprpvemenUactivities should be "home grown" and probatxly 
locally initiated:* - , ' . 



Schodla tend to make such major adaptations in externally 
developed materials that the need for external development 
may be questioned (Berman and McLaughlin, 1977? Charters 
and Pellegrin,, 1973; Stearns et al., 1977 )• 

Externally provided technical assistance is typically not 
^.positively related to school improvement -outcomes (Berman 
and McLaughlin, 1977). Xven where it is, it is much less 
important than the roles pldyed by internal change* agents 
(Miles et al., 1978). , * . . 

The organizational characteristics of schools as a class 
mitigate against effective, externally provided school ' 
improvement (Oerr, l976;^Weick, 1976) although not neces- 
sarily against more localized improvement. ■ * 

TJe organizational characteristics of aphcjuls bverwhfflm 
tf\^characterist|c^of the external intervention: local 
structure, culture and staffing/ pupiWharacteristics 
are the major determinant of innovative behavior 
;nblum and Louis, 1981; Hage and Aiken, 1970). 



• Whether or-not innovations are adopted, implemented and 
~ f maintained is not a rational, predictable process, but is 
conditioned by critical events, changes. in the process, 
"politics", and other features {March and Olsen, 1976*) 

Some of these arguments are based on the primary potency of JLocal character- 
istics; others are more related to the lack of potency of. external interven- 
tion.* In the present paper., we first examine the degree to which the ROU 

•interventions,, were potent as schdol. improvement strategies-. Second we will 
examine »the importances of looil^ effects; and " finally, we will draw some 
conclusions about effectiveness of both the intervention 'and local character- 
istics on the outcomes of Vie program. Data Jfor these' analyses are derived 
from„ a subset of up to 90 schools which participated m the program. In 
addition to survey data from 4 teachers* and principals, data sources included 
either a "mini-ethnography" or tour-five day site visits by Abt Associates 

-staff.* . • * l ' 

'0 



* For -fl n #< ten 8 iv . e description of the methodology for the study, see 
"Policy; Researcher as Sleuth," Louis, 19B1.' 

* " • ' ' « • 2"7 



Program Effects: The Power of the Intervention 

The RDU intervention contained several strategies: smaJJ amounts 
of direct funds to lo^al school sites;, technological support through the 
introduction of ex^pfnally developed programs, practices and materials; 
external human assistance to schools engaging ift a problem-solving process; 
and stimulation of required internal problem-solvingactivities. The effect 
-of each wiil be discussecPm turn- 

Money . Financial resources directly available from RDU to local 
sites were very limited: $1000-58000 per site. Project (federally^ con- 
tributed costs were but a fraction of the actual costs of* the innovative 
process* Cost data were obtained from 22 sites, through mt^Fisive examina- 
tion of rdbords; plus interviews with major participants* 

V. Van ables measured : * two types of costs were identified: 

* 

• Direct costs : Specific ROU activities paid for 
- directly with RDU grant funds, e.g., purchase of r,, 1 
the R&D product; compensation pf substitutes to 
release teachers for RDU activities, etc. 

<" • In-kind costs : Specific RDU activities not charged 

t 0 * an RDU grantr In^kind castsr are incurred when 

district funds and other non-RDU sources provide re- 
* (± source^ to the flDtf effort*, or when persflpnel time is 

J cwtributed to RDU without being directly compensated 

for by RDU program funds. 

2) Findings and discussion : In RDU, program funds accounted, on 
average, for only 20% of the local site costs of participating in thtf pro- 
gram. Thus, typically, each dollar of federal money leveraged about four 
more from the school and school district, or from other sources. 

The total, costs of the project (direct plus m-kind) and the percent- 
age of costs that' were in-kind were both correlated with five basic outcomes 
measures for the RDU program (see Table 1). The results indicate- that the 
total costs of the activities at thle sibe level are nit significantly corre- 
lated with any outcome measure (although the trend indicates that the higher 
, : the expenditures, in total, the less likely that the project Caused signifi- 
cant positive outcomes). 

The percentage of in-kind costs was a more powerful positive predic- 
tor of success. The data in Table 1 suggest that a school's commitment of 
in-kind, resources reflect or motivates a desire on the part of participants 
to achieve successful outcomes. Increasing proportions of in-kind costs weYe 
positively associated with greater organizational change, greater incorpor- 
ation of the R&D product, and more proncrunced personal impacts onTteachers. 

* 

Technological support: the impact of RftD product^ . Each project 
consolidated a "knowledge base" of externally developed programs , pract ices , 
o^ products (witji,„an emphasis on' those which had been field tested or^ val- 
idated) and which were made available, 83 appijkpnate, to local sit^s as 
solutions to their identified problem or need. _ ^ 
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Table 1 



Rank Ocder Correlations Between Costs and Outcomes 



Organizational 
Impacts (N-22) 

Incorporation of 
problem-solving 
process (N-21) 

Incorporation of 
~WcD^product {N-22) 

Problem solved (N-21) 

/ 

Personal impacts (N-21) 

> • 



• 




/ 

Total $' 


* 

% Inkind $ 


.04 

* 

no 


.49 
p-.U 2 


-•09 

NS 


• 24 

NS 


C '20 
' NS ... 


• 41 +* 
P-.06 


-.31 
NS 


* .20 
NS 




;.39 

p«.08 



23 2s 



V 



1) Variables measured : 

« 

• perceived quality of the product; 

•s perceived difficulty of implementation; , / 
v * 

local materials development; 

f • adaptation of R&D product before implementation; 

• adaptation of R&D product after implementation; 

. " V-,' « t whether, the product was field tested or validated; 

, • tKe relative* advantage of the product compared to 
prior practice; 

" * the ma&h between the defined problem and the 
\ pWfJuct; 

• the complexity of thje product; 

• the reversability of the product; and 
v • .product included adequate guidance for lmplemen- 



tdtion. 



2) f indings and discussion : Table 2 reveals that product character- 
istics are ,very powerful predictors of school-level outcomes--with the 
exception of process incorporation. The percentage of variance explained by 
three or four product variables ranges from 46S in the case of reports that 
the problem was solved] tG 105S in the case of process incorporation* The 
several variables that , enter, more than one equation show interesting pat- 
terns* Product quality ,, which reflects the degree to which teachers and 
principals 'ratp the products as relevant* .applicable to their situation, and 
providing a genuinely new way of doing things, is particularly important in 
predicting the degree to which the problem was solved* the level of program 
incorporation, and the staff development benefits reported by the teachers. 
the* complexity of the product is important in predicting overall organize 5 : 
tional impacts* program incorporation* and staff development outcomes % 
* Difficulty of implementation is a major factor in the degree to which the 
problem was solved* and the ♦overall organizational impacts* 

Product characteristics are, overall* significantly more important 
than most current implementation theories allow. Good products not only 
h^lp t6 create organizational effeets — student impacts, and organizational 
change— but ^also have significant staff development spinoffs* Local mate* 
ri^ls development and adaptation, rather than facilitating implementation 
and* institutionalization, show weak but consistently negative^ relationships 
with, outcomes* We believe, based on our site visit data that externally 
developed products can be implemented with only slight tinkering ^lf the 
school h^ carefully defined what it id it needs, and has gone through a 
systematic process to find a product that wi^l fit not only the problem but 
thfe local context, ft is not necessary ,to reinvent the whegl in each dis- 
trict in order to obtain high levels of schpol improvement . 

Process support: the impact of external human assistance * Two kinds 
' of external hfciman assistance were provided to schools through most of the ROU 
projects: th& services of a "field agent," facilitator,* or other general- 
ise who was employed by the project to support the school in its activities 
over the entire pr obLern-solv ing period; and also specialized, episodic 

ERIC • ' • ' ' • ■ i4 ' 3Q- • 
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Table. 2 



Standardised sicpvine Regression 
tween Produ 
» of School 
* <N - 60) 



Coefficients 



For the Relationship Between Product Characteristic* and Six 
/Measures of School Outcomes 



Product * 
< Characteristic 
Var i afrieg 



Product Quality 
* 

Difficulty of Implementation 

New Materials. Development 

Fre-Zmp,lementation Adaptation 

Post-Implementation Adaptation 

Product Validated 

Relative Advantage 

Hatch to Problem 

Product Complexity 

Product KeversabLli ty 1 * 
■ 

Adeq. isplem* Guidance 



Multiple R 
Adjusted r 2 



Organizational J/Product -Process Problem scope of Personal 

impacts ^ -y Incorporation Incorporation solved Implementation Impacts 



.24* 



.28** 
.17 



.31' 



.20* 

.19 > 
.29** * 



.35*' 
.20 



.34' 
•26 



.46 
.40 



.17 
£.10 



,58** 
.23* 



,19 
.31** 
-,27* 



.36** 



-.19 



.22 



-.16^ 
-.13 



.13 



.17 



.21* 



* ■ 



•51 
.46 



• 33 
•26 



• 36 

*30 -7 



' * ^ C ° ,f f l lcl,nt » * r * Panted only for those variables which cpotribgted to the reported multiple R 2 

Jle ordH r " aB / tfl 2^ wh * n «Mltlonai variable, failed to increase the Multiple R bj 1% ol more, 

the ord«r of tntry was unforced. „ . p * * ' 

/ * * . i 



* P < .05 , 
•*p < *>l 
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training which was typically intended to assist the school in implementing 
its chosen externally developed produpfc,, or ,in supplementing it with 
materials as necessary. * 

» 

1) Variables Measured ; 

a Field agent variables ; field agent initiative and 
activity, field agent time on site, field agent 
takes a political perspective on- the change process, 
field agent has an innovative personality structure, 
field agent. takes a structural perspective on the 
change process^ field agent- contact with the prin- 
cipal. ^ 

a Other consultant variables ; Amount of training;, 
diversity of training (br-numbef of sources from 
which training was provided). 

V- • . , * 

2) Findings and discussiori : . Table* 3^indicates that the external 
human assistance provided to schools can-fiave majojr^impacts upojp the degree 
to which knowledge is used and new programs lmplemeoFed. Technical assis- 
tance and "training activities have particularly potent impacts on overall 
organizational change, and program incorporation, where 36S % arw 403 of the. 
variance is explained, respectively, Only process incorporation and per- 
sonal impacts are poojly explained by the level of human assistance. (Note 
that it was also poorly explained by the characteristics of the product- 
issues related to process incorporation will be discussed in more ^detail 
later.) % . ^— — - 

Three variables stand out "as being most important, 'and of these, 
one is related to field agent behaviors, and two are related to training. 
The amount of training received by the site staff prior to implementation 
and after implementation has a strong "positive effect, and this impact is 
augmented by having training provided by a variety bf d ifferent types of 
people . * 

The time that the fietd agent spends with local site committees > 
or ('problem -solving teams" is predictive, of several dependent measures. 
Our site visit's revealed thai much of the importance of the agents can 
be attributed to the role that they" played on site in both stimulating 
conmittee members to stay active and to reach decision poants, and also in 
providing logistical support -to ensure that* the meetings were scheduled 
regularly, that suggestions for consultants were obtained, etc. Thus, the 
actual presence of the agent on-3ite was important. 

/■«•***■ 

, There is a tendency, revealed both by the quantitative and qualita- 
tive data,- for bhe two types of external human" assistan.ce to have somewhat 
different impaots on the site. Beneralists and field agents have their 
greatest /impacts in stimulating the sonool* to define their problems more 
broadly^ and to think more ambitiously about what they might do to solve 

r 

• ■ 

"Other analyses indicate that "training provided by the developer is the 
most important in producing positive^ school outcomes, but training from other 
consultants, and district specialists who have become involved either to aug- 
ment the developer or to provide specialized resources are also important. 

p ■ • • » 
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Table 3 



Standardized stepwise Regression (Beta) 4 Coefficients 
For the Relationship of External Assistance and Six / 
KeasureT'of School Outcomes ' 



t 



External * 

as 9 * m tanwv 

variables 


Organizational 
Impacts 


Product 
Incorporation 


"Process 
Incorporation 


Linking Agent (L.A.) 
initiative ana Activity 


* 




• 


L A. Tim* m cU A 


•19 


.23** 


• 


L.A. Political Perspective 








* 

L.A. Structural .Perspective 


* * 




* 


L.A* Innovative Personality 








L.A. Contact with Principals 


• 

.16* 






Amount of Training 


.33" 


.iq 




Diversity of Training Sources 
j_ : 


> ^25* 


# 

.43** 


.22* 


Multiple R 2 


.40 - 


.43 


.14 


Adjusted Multiple R 2 




.40* 


.10 











* Problem Scope of 



personal 



.24- 



,31' 



.13. 



.13 



.13 

.24** 

.31** 



26' 
.19 



.21 
.17 



.4 6 
• 41 



.26** 
.21* 

.14 



+ fete Oo.Cftel.nts are presented only for those variable, which contributed to the reported mjltlple R 2 . 



. • P S .05 
•*p < .01 
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them, thus producing a change program of greater scope* The specialized 
training from consultants , ^on -the other hand, has more impact upon the 
degree to which there -ere actual school improvement impacts within a school: 
whether the problem is ^olved, and whether there are broader organizational 
changes* * . * 

■ 

« • 
The impact of internal problem-solving activities * The ROU approach 
required the participation of local school pe*6onnel in a variety of problefti- 
sslving activities. All of the RDU projects attempted to provide structures 
and criteria for this process although tliey had less direct influence over 
the internal process than they did on the external products that were made 
available 6r on the external human assistance intervention * The" process was, 
however, air^n^ortant feature-of the RDU approach and the following features 
of the process wfere examined to. determine their impact on school outcomes*, 

r 

1; Variables Measured ; * / 

TO 

, level of effort, Quality of the problem-splving process, ' « 
, faculty influence oK the process, as well as committee ' 
or team influence, central office influence, principal 
influence, principal level of involvement, breadth of 
involvement in solution selection and breadth of in-- ^ , 
volvement in implementation* * 

* | 

2) Findings and Discussion ; The internal problem-solving process 
accounts fo0 less variance lpuour quantitative measures of school improvement 
outcomes than either the products or the external technical assistance (Table 
bj* This corresponds* also to our analysis of case data, which suggests that 
many sites arrived at "successful" school improvement outcomes via a wide 
variety of locally designed* routes* In some schools centralized decision 
making by the superintendent or principals was highly effective; in others, a 
decentralized, staff development approach worked well* Nevertheless, our, 
statistical analysis does indicate a modest level of predictive power fori 
internal process variables, particularly for the overall organizational 
change outcome, „and somewhat for process incorporation, which was not well 
explained by product characteristics or external human assistance. 

Most of the predictive power of the internal process on school 
outcomes is attributable to the breadth of involvement in solution selection , 
and implementation, and overall faculty influence QveT the decision-making 
process! [l should be remembered that breadth of involvement, in implemen- 
tation reflects not just the involvement of the faculty and the principal 

within the implementating school, but also involvement on the part of the 
si(p&fintendent , central office ^specialists, and other relevant actors* A 
high score on this variable typically represented a district in which the 
central office staff took at least some interest in monitoring the imple- 
mentation process, in providing -support, and in spreading the new practice 
to other -schools in th§ district, but did not dominate the process. 

The involvement of the whole fejculty in the problem solving process was, 
we observed on site v isits, often* a key element in spreading a "sense of 
ownership 11 from a small team or committee that designed and selected the 
innovation. Some teams were designed to increase faculty involvement, either 
by representing all grade levels or departments, bnd using representatives 
to "spread th£ word, 11 or by holding special faculty meetings to discuss and 
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Table 4 



Standardized Stepwise Regression (Beta) Coefficients ^ t < 

For the Relationship Between the Internal Problem Solvinq process v 
and six Measures of School Outcomes 



Internal 

Process 

Variebles 



Level ( of Effort % 

Quality of Problem- 
Solving procees 

Ta9ulty influence on 
Proceee 

Principal Influence 
on Process 

Superintendent Influence 
on Process 

Other Central Staff 
Influence on J>rocese 

Breadth ot n Involvement 
ia Solution Selection 
i 

Breadth of. Involvement 
In Implementation 



2. 



Multiple R" 

* 2 
Adjueted Multiple R* 



Organizational Product Process Problem Scope of , Personal 

Impacts Incorporation Incorporation Solved Implementation Impacts 
- (n-90) ' (n-90) <n«76) * ( n -76) (n-90) » (n-76)_' 



.11* 



.11' 



c 



►24* 



.23- 



.3S 
.34 



13 



.29- 



.15 
.12 



.20** 



-.20 



.24** 



.20 
.15 



.15 
.11 



.23* 

3 , 



.15 



.31- 



.16 



^T2* 



.05 
.02 



♦ Bete Coefficients are presented only for those variabiee which contributed to the reported multiple R 2 . 

The selection process was stopped when additional variables failed to increase the Multiple, R Z by 11 or moret 
the order of entry wae unforced. • ' 



* P S -05 
•*P i .01 
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vote on key decisions. Where faculty as a whole (or all of those that iould 
reasonably be affected by the planned school improvement activities) were 
regularly involved , the transition between the small group that provided the 
legwork, and the other potential usfirs was inevitably smoother. 

One of the surprises of this analysis is the fact that principal 
influence was not ja powerful explanatory factor. Our site-visit data in- 
dicate thpt, in many of the most successful schools, principals facilitated 
the process of problem definition, solution selection and implementation, but 
preferred t,o let the process be teacher dominated. Thus, while not totally 
passive, they did not tend to rpcep^e the highest scores for influence. This 
strategy, of course, worked only When there wepe active faculty who rfere able 
to take on leadership rples 'in promoting the process. 

A final surprise is that the internal problem-solving process does 
not predict^ the level of staff development benefits reported at a school. 
Based upon bQth theory and at least some of our site visits we would have 
predicted. t£at staff development benefits would have been more strwgly 
associated with process variables .such as level of effort and faculty in- 
fluertte. However, staff development outcomes, at .least as they are aggre T 
gated to the school level, are largely a function of the amount of training 
recefved by staff roembexs.\ UJtbex analyses presented elsewhere suggest that 
staff members who are on the team do derive substantially greater staff 
development benefits than those who are not on the team, indicating that, for 
individuals who are most involved, the process may make a • difference. ) 

A * 4 

The Combined Intervention? Products, Processed People ^ * 

The previous sections examined the impact of each aspect of the 
,intervent ion separately. Overall, 'the product characteristics and external 
human assistance each separately explained greater percentages of vanafce 
in school outcomes tfcan did the lhternal problem-solving activities. Not 
surprisingly* incorporation of the problem-solving process was the only 
outcome that was affected more "by! the internal problem-solving activities 
than either of the two external interventions, although the adjustedi multiple 
R2 /was not very'great (-15). ■ - * / 

< V 

However, the ifrpact of the,R[Al intervention cannot bg understood by 
only examining the three intervention strategies separately. In reality the 
intervention combined the thiee strategies, and it is therefore important to 
examine £he potency of the combined approach. In order to do so, multiple 
regressions of outcomes on a s$t of< independent variables drawn from each of 
the three intervention strategies <were .conducted. The following variables 
teach of which was a powerful predictor within its own group) were chosen: 



/ 



• 



product variables : product quality, product complexity 
product validated, and difficulty of implementation; 

external human assistance variables : field agent/ , 
principal contadt, amount of training received, diver- 
sity of training, and fi61d agent time on site; 

% 

internal problem solving process variables : faculty 
involvement in the process, breadtVi of participation 
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in solution selection, breadth 6f participation in ^ 
implementation and the quality of the proBlem-solving 
process, ^ ' 

Table 5 indicates that the real potency of the intervention is a 
function of the combination of strategies, resulting in high or very high 
percentages of variance 'explained on each- of the school outcomes. For ex- 
ample, adjusted multiple fTs were well over for organizational impacts 
and for product incorporation^ Even process incorporation, the most "elu- 
sive" of the school outcomes in our analyses had 24S of the variance ex- 
plained by a combination of six variables drawn from all of the intervention 
categories, 

V ' ' - i' *~ 

The most important predietor variables of the combined iintjsrvent ion 
strategies are product quality (which enters into the equation for each 
outcome;, product characteristics sucf» as complexity and prior validation, 
amount of training received , faculty participation in the process, and 
breadth o f participation in solution selection and implementation'. 

Product characteristics and diversity ofr training appear to be 
particularly important "to product incorporation . But ironically product 
quality and prior validation are negatively related to process incorporation. 
The only variable that lsj^ignificantly positively associated with both 
program outcomes is diversitjp'f training sources. This, suggests that it may 
be extremely difficult to have both objectives in the same program. Both, 
site visit and survey- data suggest that the program was more successful at" 
achieving product incorporation and spinoff effects' (organizational changes 
and staff development effects) than process incorporation. (Note this is not 
the same as a high quality problem-solving process, which many sites did very 
well as participants in this program.) 

It is particularly interesting that, for each outcome, the variables 
that contributed to the explanation of the outcome were drawn from each of 
^the three intervention strategies. Furthermore, with .the except ion~5T~one 
outcome, a combination of intervention strategies is a more powerful predic- 
tor of the outcome than any of the individual intervention categories (see 
Table 6). The one exception is the outcome of "problem solved" which" is 
predicted better by product variables (RS.46) than by a combination of 
strategies (R =.41). ■ - 

The Impact of Local Site Characteristics 




- i • — — . . ,w vim uiw iivu aLiiuui uuiluiiicS) anu to 

compare those results with the impact of the intervention, several analyses 
were conducted. ai.a* 7 oco 

Variables were measured in five categories: 

• principal characteristics ; how long in the, school; 
teaching experience, administrative experience, and 

^ degree to whiqh staff rate him/her as an instroction- 

* al leader; , 

• * 
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, Table 5 



Standardized Stepwise Regression (Beta) Coefficients 
For the* Relationship Between Combined Intervention Strategies. nnd 
Six Measures of School Outcomes 
<N«75> 



v 



Intervention 
Strategies " ~ 

- - - 

{Product) 

Product Quality 

Product Complexity 

Product Validated 

Difficulty of Implementation 

(External Assistance) 

Linker/Principal Contact 
Amount of Training 



Diversity of Training 
Sources 

Linker Time on Site 



(Internal Problem-Solving 
Activities) " 

Taculty^ Involvement 

Breadth of Involvement 
in Solution Selection 

Breadth of inv. in 
^Implementation 

Quality of Process 



Multiple R 

Adjusted Multiple R 2 

9 



Organizational froduct Process Problem 

Impacts Incorporation Incorporation Solved 



Scop* of Personal 
Implementation Impacts 



.18* 
.29** 



.17** 
.22** 



.09 



.16 



.11* 



.59 



.55 



.12 

.15* 

.18* 



.14 



.21* 



.56 
.52 



-.20* 



.27** 



.18* 
.23* 



.09 
.20* 



.30 
.24 



.58* 



.13 



.22* 



.20* 



.21* 



.37** 



v * 



.16 
ft 



-30** 
.2R** 



.13 

* 

.22** 

on** 



.08 



17 



.4 3 
.41 



47 
.43 



♦ 42 
.36 



^RJ^G * Btt* Coefficients are presented only for those variables which contributed to the reported multiple fC 

Ihe selection process was stopped when additional variables failed to increase the Multiple R ^by II ot more 
^the ocder of entry was unforced, 1 * 



Table 6 




Percentage of Variance In Outcomes Explained by Three Strategies 
of the intervention and the Cocibined Intervention Strategies* 

(H - 75) 



Predictor 
Variables 


Organizational 
Impacts 

* 


■ Product > 

Incorporation 

> 


Process 
Incorporation 


Problem 
Solved 

i 


£cope of , 
.Implementation 


Personal 
Impacts 


Product Characteristics 


■ .28 ^ 


.40 'f 


.10 


1 

.46 r 


.26 


.30 


* 

External Assistance 


\ 


.40 . 


m 

.10 




.41 




Internal Problem Solving 
Activities 


.34 \ 


.12 

¥ 


.15 


- .11 


.12 


• 02 .\ 


Combined Intervention 
m Strategies 


.55 \ 

< \ 


* 

.52 

4 

\ 


.24 


.41 


.43 


.36 > 

• 



♦Adjusted multiple R v 
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• teacher characteristics : percent male; percent teach- 
ing tor ten or more years in the schaol; average num- 
ber of professional memberships; percent with an'ad- 

* vanced degree; jp ^ 

• school siae, structure and climate : size of district f m 

♦ sizg of schoo^, influence of principals, teachers and 
superintendent over key educational decisions; school 

i level (elementary or secondary)*, staff orientatioVi to 
change, collegiality , tension among staff, previous 
experience with similar problem^solvmg activities; 

r ^ s T • . * 

• characteristics -of the community setting : index of ^ 
disadvantagement among students, % students 'frort white % 

collar' families, level of> community change, Yurality, 

• * * 

• nature of the problem s magnitude of problem, focus on 

classroom organization, ftfcus on curriculum? or mate- 
rials, focus on pupil perfonjianc^, focus on rolfe, rela- * 
tions, focus on school organizational problems, focus 
on problems in staffing o{ staff characteristics, 
/ocus.on pupil attitude's and behaviors. » 

Findings and discussion . The results of regressions* of outcomes 
on each of tteese categories separately had little explanatory -power . For 
both principal characteristics and characteristics of the community! seating , 
there .were no regressions that explained as much as 15% of the . variant in 
any dependent variable. For teachers characteristics, only percentage of 
staff who are male contributed significantly to the explanation of. overall 
organizational impacts* It is jmter-es^ag to note that this relationship was < 
a negative -one, suggesting that mal# teachers (who were also more typically 
•in secondary schools J may be particularly "independenC'Vand resisttfht to an 
external intervention .and the kinds of collaborative efforts that were -a 
feature of the RDU program. Three structure and climate variables did 
expiain 15% of the variance in overall organizational impact: teacher change 
orientation , principal influence pver decision making , $nd teacher influence 

over decision making * The ,only - hategory of site variables that explained 
three outcomes (.organizational impacts, the degree to which/the problem was* 
reported to be solved, and personal and staff development* impacts) was 
characteristics of the problem that the sites dealt^ with in the program. The 
most important variables were a focus on classroom organization and pupil 

performaQce . ( , 

* -However, oner'further step was taken, which was to examine the com* 

bined impact of the most potent site variables (based on simple correlatdons 
as well as the regression analyses) on the -school impacts* For this analysis 
the following .variables were chosen; teacher orientation to change and 
teacher* influence over decision making; the inderx of disadvantagement of 
students ;- school level; percent male staff; the degree to which the problem- 
solvipg* activities had begun prior to the RDU program (an index tff "readi-> 
ness"); and the identification of the problem aa being one of classroom 
organization or pupil performance* As Fable 7 shows, these variables do' 
explain h relatively high percentage of^ variance on many of the outcomes, 
particularly "product incorporation IR =.45) and organizational impacts 
(R =^ ,40)* Personal* impSfcts are explained leas^ well by aite characteris- 
tics* Once aqh\n a highly potent variable (identification of the problem as 
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one of pupil performance) was negatively related to process incorporation. 
Other- variables that were predictive of both product incorporation and 
process incorporation are the degree of teacher influence ^n. dec isj^flr making ,^ 
and the indicator of readiness* 

The Relative Impact of the Intervention and Local Site Characteristics 

A major objective of this paper has been to examine the relative 
potency of the intervention as compared to the site characteristics on the 
school improvement outcomes. While si,te characteristic^ proved to be power* 
ful predictors of school outcomes,. Table 8 indicates that for all but «one 
outcome measure x the power of 'the intervention outweighs, local si^e charac- 
teristics in explaining the outcomes. 



We interpret this as implying that the fiOU intervention 'was partic'u-. 
larly effective in addressing the inequalities in innovat i veness among 
schools that naturally occur as a result of differences in personnel re- 
sources, community resources, prior innovative experiences, etc. (In Fact, 
there was no significant difference in outcomes based on school size, level, 
rurality or community turbulence, whereas the index of disadvantagement was 
positively correlated with outcomes.) The biggest difference Is in |he 
adjusted R for personal staff development outcomes (R = .36 vs. R = 
.16), followed by an effect on organizational changes (.55 >vs. .40). In 
other words, the spinof f\ef fects of the program were most markedly affected 
by the intervention. 

Only process incorporation was equally affected by both the interven- 
tion and the site characteristics, and in v each case only Zh% of the variance 
was explained by each category. How can one explain 'the relatively low 
, impact of the intervention on. process incorporation? While most of the ROD 
projects had stated objectives of permanent improvement in the general 
problem-solving capabilities of the schdol, case study and site visit data 
reveal that in fact the primary emphasis of the intervention was to proAde 
assistance for engaging in a specific, targeted problem-solving process that 
focused on adopting and installing a new product or 9 practice to splve a 
particular problem. While some training in the generic group process or , 
problem-solving skills was included, for most sites it seemeJ*hard to con- 
centrate on the capacity building function at the same time as effort was* 
being expended to solve a particular problem. Furthermore, the fielchagent 
or facilitator was viewed as jcfucial to the process, and withput sp££ial 
project support, was not likely to be' available to the local site aga)in. 

* . 

A final analysis was conducted to determine "whether site character- 
istics add to the power of the intervention in explaining school* outcpme?. 
Stepwise regressions of outcome measures on variables representing a com- 
bination of each aspect of the intervention (products, external human assis- 
tance, and internal problem-solving activities) and potent site characteris- 
tics were conducted. As Table 9 demonstrates., for all outcomes, explanatory 
power is increased when variables frort all of the above domains are con- 
sidered. Eight variables explain 685 of the variance in organizational 
change, and once again process incorporation is the most elus*ive, with 29% 
of the variance explained. It is particularly interesting to f note that for 
four of the six outcomes, the variables contributing to the adjusted multiple 
R are drawn from ail the domains of the mtervention (products, external 
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assistance, and internal proceas) as well aa site characteristics. The. 
exceptions are the degree to which the problem was perceived as solved, in 
which no variable representing" external hunan assistance had entered at the 
point in which the selection was stopped, and process incorporation where the 
explanatory variables only represent the internal problem solving activities 
and site characteriatics. * In this analysis, no variables representing the 
external product characteristics or external hunan assistance contributed to 
the explanation of process incorporation. This analysis reinforces the 
interpretation that the intervention may' not have been successfully fostering 
the incorporation of improved problem solving. Instead, the degree to which 
the outcome was achieved was largely a function of the internal processes 
(which were less influenced by the project than th^ external features df the 
intervention), and the less manipulable site characteristics themselves. 

*»*###***## 

t 

Conclusion 

* * 

The previous two papers have related* part of the Qoraplex story of 
how schools participate in the RDU program were affected by its relatively 
unusual aasistance strategies. A more detailed discussion of the policy 
' implications of this study are presented elsewhere (Louis and Rosenblum, 
1981) but ,a final distillation of what has been learned would emphasize the 
following points: ^ p ' 



\ • Dissemination programs create two types of outcomes 
at the school level: knowledge utilization/imple- * 
mentation and school improvement/capacity -building} 



Engaging in a broad knowledge utilization activity 

is one of the most effective means of building capacity; 

Good products produce gtfod school outcomess quality, 
control is a critical element of an effective'dissem- 
mation strategy; ✓ 

External technical assistance is important to facili- 
tate both knowledge utilization and school improve- 
ment. On the whole, training provided by experts and 
program developers that related directly to knowledge 
utilization objectives was more important than general- 
lst field agent support in producing both knowledge 
utilization and capacity building improvements; 

Field agents '(generalists) were important in facilitating 
improvements in problem-solving behaviors at the school 
level, and increasing the level of effort and scope of 
knowledge utilization. However^ a high level of involve- 
ment by such agents may diminish. capacity-building out- 
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The quality"of the problem-solving process is leas im- 
portant in producing knowledge utilization outcomes, 
than has often .been thought. However, it is a key . 
to other school improvement outcome^; 
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School characteristics suqh as the staYf's orientation 
to change and the amount of principal influence are 
important determinants of how well schools *wiT$ im- 
plement a problem-solving process, but they do not 
.overwhelm the impact of the intervention; , 

The biggest payoff in. terms of both knowledge utiliz- 
ation and school improvement will be realized by em- 
phasizing the resolution of problems that affect the 
core activities of the school — teaching ^and pupils; 

Costly planned change efforts are no more likely to have 
significant impacts*on the school* than less expensive 
ones* However, it is important' to allocate a large 
proportion of the available resources to pay for staff 
involvement in selecting a solution and planning for 
implementation* ,It is also important to supplement 
external funding with internalTy contributed Staff 
time and other r£source&; and , 

• * * 

While not all schools followed program specifications 
for a rational problem-solving process and the imple- 
mentation of an R&D-based, validated "pred&pt," the 
program intervention had almost no significant nega- 
tive impacts on schools that might offset the gener- 
ally positive findiqgs presented above* 

v 
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• ■ ANNOTATED & BLIOGRAPHY OF 'REPORTS 

FROM THE STUDY (JF LINKING R&D WITH SCHOOLS 



Chabotar, K.J. and D«G. Keil. Linking R&D with . Schools: An NIE Program and 
Its PoUcy Context . Septercbe r ,^978. 18 ppT * 

The report presents a brief overview of the R&D Utilization 
- program, and a more 4H&led presentation of the policy 
v questions that* the study of the program will address. The 

rankings that state and federal policy makers attach to the — 
various policy questions that form the basis for the study 
are discussed. * 




Louis, K.S., J. Molitor* G. Spencer, and R. Yin. Linking R&D with Schools : 
An Interim Report* Septerrbef, 1979. 39 pp*^ 
> 

The report presents a description of the R&D Utilization 
program,, and the sevefi gperating demonstration projects* 
Characteristics of the projects that are common to all, 
and those that are distinctive are identified. Prelim- 
inary observations about the natVre of services being 
delivered to schools and the lmpacTs of these on school 1 
improvement activities are discussed. Several vignettes 
of school activities injthe program are p>esented. 

Kell» D. and K*S. # Louis (with S, RoSeriblum and X.A. Molitor)« The Role of 
Local Action Teams in School Improvement . Junfe, 19_80 v 43 pp. 

The report focuses on ai major objective of the RDU program: 
to increase participatory decision making in school^. The * 
ain) of the report is to provide teachers and administrators 
1 with guidelines for establishing effective problem-solving 
- teams* The conclusions* of the report are illustrated by 
fche experiences of thrpfc very different schools that were 
Involved with the program* * * 
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Louis, K.S. "Linking R40 with Schools: Products and Processes: Some Prelim- 
inary Findings from the R&D Utilization Program and Their Implications for , 
Federal Dissemination Policies," f>aper presented at the 1980 meetings of the 
American Educational Research Association, 26 pp. 



The paper presents a preliminary analysis *of the survey 
data from 90 intensively studied schools. The paper 
concludes, on\he basis of) regression analyses,* that^ll 
components of tfies^Uanoa^RDU intervention strategy— the C 
use of high quality "products," the application of technical 
assistance from external field agents and trainers, and the 
guidance of the school through a rational, participatory 
problem-solving process—hdfre a strong impact upon knowledge 
utilization processes and outcomes. In addition, the 
report concludes that the effects of the variables measuring 
RDU strategies outweigh characteristics of the school* such 
as readiness to engage in a change. program. 

Louis, K.S, "Linking R4D with Schools: Implications for School Adminis- 
trators from the Study of the R4D Utilization Program," Paper prepared 
for th6 1980 Summer Instructional Leadership Conference of the AASA, 11 pp, 

* t Based on data from preliminary^ analyses of the impact 

of the RDU program at Jthe'scKbol level, several recormnenda- 
tions for how schooliidministrators may facilitate the 
problem-solving process in schools are dfrawn. These 
include the need to emphasize using 'externally devel- 
oped products where* they are availably and appropriate, 
# attempting to maximize a change effort by encouraging 
the adoption of complex new # practices, and the importance 
of adninistrative support ui the continuation and incorpor- 
ation phase. Other recommendation^ include the importance 
of promoting teacher-driven, participatory dhange teams, 
find searching for external facilitators who can provide 
assistance and stimulation to a locally driven process. * 

Spencer, G, J, and K,S. Louis (with S, Rosenblum and c/Takata), Special 
Report on the Training and Support o f Educational Linking Aqents- Septerrber. 
1980. 63 pp, ' 

, The objective of the report is to describe and assess 

the types of training and support that were provided to f 
field agent personnel in the RDU program. This investiga- 
tion is based on data provided by the RDU prdjects on the 
content, timing and methods of training activities for 
field agent personnel, and surveys of 49 field agents who 
were employed by the program for two or more years. In 
addition, "gupoori," or ^informal communication, supervision 
and technical tssistarfceT to field agents are also analyzed, 
tiding the'same day^sotirces", and supplemented by 1 interviews 
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with field agents' direct supervisors/ The report con- 
cludes that (1) there were only minor differences in the 
formal training opportunities that eaCh project provided to 
field agents; (2) afield -agentj^gene rally would have pre- 
ferred more variety in contrfit and training model; (3) both 
projects ^id and agent "tvJst M organizations are important 
sources of support for agents; and (4) support activities * 
have more impact on linker self-reported behavior than 
training. 

* 4 

Yin, R.K., M. Gwaltney, and K.S. LouJfs (with S. Rosenblum). Quality Control 
andjProduet Information Systems : An Interim Report on Implementation, Use 
andlEffects in the RAD Utilization Program , October, 1980. 60 pp. 

* 

The report presents, an analysis of- the processes of dev-^, — . 

^^^Neloping and operating "knowledge bases" or pools of curri- 
c&hifl^and inservice materials that were used by the seven 
RDU pc%jMts in providing services to their client schools. 
7h& majorVssues addressed include those of locating^ 
* , acquiring and certifying materials, and the problems of 

matching locally defined school needs with the information 
that was available. The analysis indicates that despite 
considerable efforts on the* part of N1E and the seven 
' projects to emphasize the dissemination of validated R&0- 
based products, as many as 60S of the products adopted 
by the schools did not meet the criteria established in the 
program design. Some reasons Tor the discrepancies between 
intent andi implementation are discussed. 

■* 

Corwin, R.G. Program Design and Implementation: Biography of a Federal Pro- 
gram lrwits Funding Agency? November 3, 1980. 90 pp. 

This reports analyzes the role of HIE as an agerfcy in 
stimulating and, supporting the development of {he RDU 
program. The report emphasizes the interaction between* the ^ 
«j — structure of ^he agency and the evolving events as they 
shaped majoj program decisions. 

Chabotar, K.J. f K.S. Louis, and S. Rosenblum, "The RDU Study and Its Policy 
Context: Perspectives of Educational -Policy Makers/' December*^ 1980. 
26 pp. 

This memorandum to NIE summarizes some of the preliminary 
findings Frofn the RDU program, and the reactions to them 
of 14 major educational policy makers in the National 
— Institute of Education and the Office of Education. « 



" 52 



Louis, K.S., S. Rosenblum, and 0. Kell. "Staff Development and Curriculum 
Change * What's Good for Teachers is Good for Schools." February, 1981. 

^ 

This brief report is intended to communicate to teachers 
the, findings of the study regarding the staff develop/tent 
benefits that occurred as a result of participating in 
the ROU 'process. The report concludes that teachers who 
participated'on a team benefitted more than those who did v 
not, thatr>providing expert training i n imple menting a , 
new curriculum produced more staff development benefits, 
and that focus on practical classroom problems was bene- . 
ficial for teachers. The report concludes that merging 
inservice/staff development programs and planned change 
progr&ms*lll create a'more conjplimentary use of limited 
school funds. 

Louis, K.S. "Policy Researcher as Sleuth: Integrating Qualitative and 
Quantitative Data." p ape r presented at the 1981 meetings of the American 
Educational Research Association. 25 ^pp. 

• This. paper presents an overview of the methodology of 
the study of the R&D Utilization program, and discusses 
- stheuseof a "consolidated coding" approach to merging 
datansjllected by sur*^y with that collected through 
semi-structured s^te visits. Some of the issues and 
problems associated w,ith the methods are presented. 

Chabotar, K.J., K.S. Louis and J. Sjogren. "Relationships Between Local 

Contributions and the Success of a Federal School Improvement Proqram." 

Hay, 198J. 6 pp. **~* 

. 

This report presents very briefly the results *of one 
component of a study of the costs of participating in 
The findings indicate that the total cost of the 
e effort (federally contributed plus locally con- 
tributed costs) is not related to the level of success 
of the change effort. The percentage of costs that re- 
presented locally contributed time and dollars was, 
however, positively correlated with success. 
* - ■ 

Louis, K.S. and D. Kell (with A. Young, G. Spencer, R. Carlson and B. Taylor) 
The Human Factor in Qisseminationi Field Agent Rol es in their Organizational 
Context . July. 1981. 136 pp. ~ \ ; f 

This report presents an exploratory analysis of the 

field agent r ole b $sed on surveys and interview 

materials from field agents in the ROU program. The . • 

report focuses on the role dilemmas of field agents, 

which include role conflict, role arrbiguity, margin- 
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aloty^-and unclear specifications for behavior* Factors 
that affect both field agent job satisfaction and job 
performance, as measured by client satisfaction and agent 
assessments of school outcomes, are discussed* Three case, 
studies that illuminate some of the problems of managing 
field agents, of developing appropriate role definitions 
with clients , and of choosing day-to-day tactids of role 
enactment are presented to enhance the quantitative survey 
findings.* A preliminary model to explain agent job-related 
attitudes and role performance is aefived from the analysis* 



Louis, K.S., D* Kell, K* Chabotar, and S*D*Sieber (uitrf P* Desmond) (eds*V 
Perspectives on School Improvement; A Casebook for Curriculum Change * 3uly, 
198T. 260 pp* 1 

This report presents a framework within which administra- 
. tors and curriculum coordinatqrs can interpret various s 
, problems in managing change* Chapter-length case studies 
of schools that participated in the RDU program are pre- 
sented, to illuminate change management issues in three 
areas: leadership and participation; strategies, and tac- 
1 tics of initiating and implementing new programs; and 
, managing contingencies that arise in the change process. 
Each chapter is accompanied by questions that are suitable 
for grqup discussion of >the case, and a chapter synthesiz- 
ing across all 12 cases is presented. The report i£ in- 
tended for use either as a text or as a book of readings 
for school professionals* 




Louis^K*S* and S. Rosenblum (with G. Spebcer; J. Stookey and R. Yin)* 
Designing and Managing* Interorqamzational Networks * 3uly, 1,981. 180 pp. 

* 

The report examines the assumptions underlying the 
* cfmphasis of the RDU program on the development of mter- 

organizational networks to support the delivery of mform- 
* at ion and technical assistance to schools. A model for 
examining network design," network manage mept^ and the 
outcomes of networking is presented and illustrated through 
the presentation of four chapter-length cases of RDU c 
-programs. The final chapter derives conclusions and 
recommendations regarding design and .management of mter-^/ 
organizational relationships. 

louis, K.S., A. Rosenblum, arid 3. Holitor (with K. Chabotar, D* Kell and R* 
Yin)* k Strategies for Knowledge Use and School Improvement * July, 1981. 
260 pp. ^ ^ 

The report examines the' process of cfiange at the school 
level, using a framework that draw? upon current organiza- 
tional theory, and assumptions about knowledge utilization 
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and school lmprovefhen^T The report draws most* N?avily ^ 
off quantitative dafea/gources to illuminate the rdlation^ip 
betwee^^^e* interven\ion4fetrategies used by the T?DU projects, 
and school processes, amf outcomes; it also uses qualitative 
cbse material to expandSjpon the findings. Jhe final 
chapter prese'nts some reflections drawn from observations 
of the participating Schools about th& ways in which 
knowledge ".utilization and general school improvement * 
can be facilitated. 



Louis t K.S., Roseriblum, S. and Molitor, 3. Strategics for Knowledge Use and 
School Iff»provement; A Summary , July, 1981. * 50. pp. \ \ 

This % ^eptrt coebmes two papers presented at the American 
\ Educational Research Association meetings in 1981 • .The 
* - first paper presents an overview of ttie outcomes of the* 
ROU program at thd school level, while the second presents 
ari^&nalysis of the way. in whicfi product characteristic/, 
technical assistance, the internal ".problem-solving prof 
and school and pupil 'characteristics predict the'Jtey^l 
of success of the program. (This report is a^surmrfry 
-of *VoJ,ume 2 of the Firial Report.), >^ 
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A Summary of 
, July, 1981 • 



This, report -served as art executive summary foi the project, 
and s^nthds'izes'the.'main findings of both volumes of 
t\ie final -rfeport in the context of .some tff the major 
policy and management decisions that currently. fac^ dissem- 
lqation programs ab the federal and 4 state 1 level. 



■ V 




t. 



So 



9 

ERIC 



ft* 



